# Inference Acceleration

FastVideo is an open-source framework designed to accelerate large video diffusion models. It offers two consistency distillation video diffusion models, FastHunyuan and FastMochi, achieving an 8x increase in inference speed. FastVideo introduces the first open video DiT distillation recipe based on PCM (Phased-Consistency-Model), supporting distillation, fine-tuning, and inference for state-of-the-art open video DiT models, including Mochi and Hunyuan. Additionally, FastVideo supports scalable training using FSDP, sequence parallelism, and selective activation checkpoints, as well as memory-efficient fine-tuning with LoRA, pre-computed latent variables, and pre-computed text embeddings. Ongoing development is highly experimental, with future plans to introduce more distillation methods, support additional models, and update the code.

Star-Attention is a novel block-sparse attention mechanism proposed by NVIDIA aimed at improving the inference efficiency of large language models (LLMs) based on Transformers for long sequences. This technology significantly boosts inference speed through a two-stage operation while maintaining an accuracy rate of 95-100%. It is compatible with most Transformer-based LLMs, allowing for direct use without additional training or fine-tuning, and can be combined with other optimization methods such as Flash Attention and KV cache compression techniques to further enhance performance.

MInference is an inference acceleration framework for long context large language models (LLMs). It leverages the dynamic sparse characteristics of LLMs' attention mechanisms, significantly enhancing the speed of pre-filling through static pattern recognition and online sparse index approximation computation. It achieves a tenfold increase in processing 1M contexts on a single A100 GPU while maintaining inference accuracy.

AsyncDiff is a method for accelerating diffusion models through asynchronous denoising parallelization. It divides the noise prediction model into multiple components and distributes them across different devices, enabling parallel processing. This approach significantly reduces inference latency while having a minimal impact on generation quality. AsyncDiff supports a variety of diffusion models, including Stable Diffusion 2.1, Stable Diffusion 1.5, Stable Diffusion x4 Upscaler, Stable Diffusion XL 1.0, ControlNet, Stable Video Diffusion, and AnimateDiff.

BiTA is a bidirectional adjustment method for large language models. It accelerates large language models through simplified self-regressive generation and draft candidate verification. As a lightweight plug-in module, BiTA can seamlessly enhance the inference efficiency of existing large language models without requiring additional auxiliary models or incurring significant additional memory costs. After applying BiTA, LLaMA-2-70B-Chat achieved a 2.7x speedup on the MT-Bench benchmark. Extensive experiments have shown that our method outperforms the state-of-the-art acceleration techniques.